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ABSTRACT 

A fixed-based simulation experiment was conducted in NASA Langley Research Center’s Integration Flight Deck 
simulator to investigate enabling technologies for equivalent visual operations (EVO) in the emerging Next Generation 
Air Transportation System operating environment. EVO implies the capability to achieve or even improve on the safety 
of current-day Visual Flight Rules (VFR) operations, maintain the operational tempos of VFR, and perhaps even retain 
VFR procedures - all independent of the actual weather and visibility conditions. Twenty-four air transport-rated pilots 
evaluated the use of Synthetic/Enhanced Vision Systems (S/EVS) and external Vision Systems (XVS) technologies as 
enabling technologies for future all-weather operations. The experimental objectives were to determine the feasibility of 
XVS/SVS/EVS to provide for all weather (visibility) landing capability without the need (or ability) for a visual 
approach segment and to determine the interaction of XV S/EVS and peripheral vision cues for terminal area and surface 
operations. Another key element of the testing investigated the pilot's awareness and reaction to non-normal events ( i.e 
failure conditions) that were unexpectedly introduced into the experiment. These non-normal runs served as critical 
determinants in the underlying safety of all-weather operations. Experimental data from this test are cast into 
performance-based approach and landing standards which might establish a basis for future all-weather landing 
operations. Glideslope tracking performance appears to have improved with the elimination of the approach visual 
segment. This improvement can most likely be attributed to the fact that the pilots didn’t have to simultaneously 
perform glideslope corrections and find required visual landing references in order to continue a landing. Lateral 
tracking performance was excellent regardless of the display concept being evaluated or whether or not there were 
peripheral cues in the side window. Although workload ratings were significantly less when peripheral cues were 
present compared to when there were none, these differences appear to be operationally inconsequential. Larger display 
concepts tested in this experiment showed significant situation awareness (SA) improvements and workload reductions 
compared to smaller display concepts. With a fixed display size, a color display was more influential in SA and 
workload ratings than a collimated display. 
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1. INTRODUCTION 

The U.S. air transportation system 1 is undergoing a transformation to accommodate a projected threefold increase in air 
operations by 2025. One of the key capabilities envisioned to achieve this Next Generation Air Transportation System is 
the concept of equivalent visual operations (EVO). EVO implies the capability to achieve or even improve on the safety 
of current-day Visual Flight Rules (VFR) operations, maintain the operational tempos of VFR, and perhaps even retain 
VFR procedures - all independent of the actual weather and visibility conditions. The Integrated Intelligent Flight Deck 
(IIFD) project 2 , under NASA’s Aviation Safety Program (AvSP), is investigating the use of synthetic and enhanced 
vision (S/EV) as enabling technologies to meet the challenges of EVO, creating a virtual visual flight environment for 
the flight crew, independent of the outside weather and visibility conditions. 

Synthetic Vision (SV) is a computer-generated image of the external scene topography, generated from aircraft attitude, 
high-precision navigation, and data of the terrain, obstacles, cultural features, and other required flight information. 
Enhanced Vision (EV) is an electronic means to provide a display (typically on a head-up display, or HUD) of the 
external scene by use of an imaging sensor, such as a Forward-Looking InfraRed (FLIR) or millimeter wave radar. 
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Prior research’' 4 has shown that significant improvements in situation awareness (SA) without concomitant increases in 
workload and display clutter could be provided by the integration and/or fusion of SV and EV technologies for the pilot- 
flying (on a HUD) and the pilot-not-flying (on a auxiliary display) during reduced visibility approach operations. The 
fused imagery showed unadulterated SV imagery on the initial approach. At 600 ft height above touchdown (HAT), the 
SV image begins transitioning to an EV image by slowly fusing the SV and EV concepts. By 500 ft HAT, the fusion 
process was completed and the resulting image was pure EV. The Pilot Flying (PF) followed existing FAA operating 
mles (Federal Aviation Regulation part 91.175) for using EV on a HUD, which require a transition from EV imagery to 
natural vision by 100 ft HAT to continue the landing. In that experiment, crews effectively used existing EV procedures 
in their decision making, but found the required visual transition from an excellent EV image to natural vision by 100 ft 
HAT to be awkward. Pilots commented that the EV imagery on the HUD provided suitable visual references to 
complete the flare and landing. 

In addition to EVO, NASA is also investigating the use of vision technologies to overcome visibility limitations found in 
conceptual low-boom supersonic aircraft designs (Fig. 1). As is evident in Figure 1, pilot forward visibility using 
forward-facing windows would be severely compromised as a result of the vehicle configuration. 



Figure 1. Conceptual low-boom supersonic aircraft configuration. 

Under the Fundamental Aeronautics (FA) Program, Supersonics project, NASA is performing fundamental research, 
development, test and evaluation of flight deck and related technologies to support these low-boom, supersonic 
configurations by use of an external Visibility System (XVS). XVS is a combination of sensor and display technologies 
which can provide an equivalent level of safety and performance to that provided by forward-facing windows in today’s 
aircraft. 

Without XVS, the economic viability of a low-boom supersonic aircraft is questionable, particularly since the lack of 
forward visibility by the pilot would severely restrict aircraft operations and airspace usage especially when the weather 
is clear and visibility conditions are unrestricted - i.e., when Visual Meteorological Conditions (VMC) prevail and visual 
approaches are possible. Without XVS, the crew of a low-boom supersonic aircraft cannot “see-and-avoid.” 

Significant research 5,6 was conducted under NASA’s High Speed Research (HSR) program during the 1990s on the 
design and development issues associated with an XVS for a conceptual high-speed civil transport aircraft. What 
emerged from this research (and still holds true today) is that the key challenge for an XVS design exists during VMC 
operations when it is assumed that the flight crew has natural visibility (even though they may be operating on an IFR 
flight plan). 

Paradoxically, EVO technologies strive to create equivalent VFR operations for today’s aircraft when flying in actual 
Instmment Flight Conditions (IMC) whereas XVS technologies try to enable an IMC cockpit (i.e., low-boom supersonic 
aircraft) to operate under VFR. The primary area of research overlap concerns the vision and visibility requirements to 
enable EVO or XVS. That is, what information and in what manner should this information be delivered to the flight 
crew to conduct EVO, whether EVO is created on the “windowed” flight deck when flying in actual IMC or whether it is 
created because of intentional restrictions to the flight crew’s external visibility due to aircraft structure and design. 

It has been hypothesized that the use of advanced vision technologies (such as S/EV and XVS) on conformal, head-up 
displays can provide precision approach, landing, and taxi guidance for “all weather” capability to all runways without 
requiring extensive approach lighting systems, ground-based precision guidance systems such as the instmment landing 
system (ILS), or other airport infrastructure. These technologies may provide “equivalent vision” capabilities which 
would obviate the need for airport lighting and other infrastructure to support the flight crews’ need to visually acquire 
runways and taxiways using normal vision. 



An experiment which combined NASA flight deck research objectives from the IIFD Project/Crew-Vehicle Interface 
Element and the Supersonics Project was conducted to investigate the use of S/EV and XVS technologies as enabling 
technologies for future all-weather operations. The experimental objectives were: 

• To determine the performance and safety of these advanced vision technologies for all weather 
(visibility) landing capability without the need (or capability) for a visual approach segment 

• To evaluate if performance and safety for all weather landing capability is affected by whether the 
head-up display of information is provided using a collimated or un-collimated display 

• To assess the interaction of head-up display information and the presence or absence of peripheral 
vision cues. 

2. METHODOLOGY 

2.1. Subjects 

Twenty-four pilots, representing 6 airlines participated in the experiment. Each evaluation pilot (EP) had previous 
experience flying Elead-Up Displays (ElUDs). The EPs had an average of 1788 hours of HUD flying experience and an 
average of 20.4 years and 13.8 years of commercial and military flying experience, respectively. 

2.2. Simulation Facility 

The experiment was conducted in the Integration Flight Deck (IFD) simulation facility (Fig. 2) at NASA Langley 
Research Center (LaRC). The IFD emulates a Boeing B-757-200 aircraft and provides researchers with a full-mission 
simulator capability. The cab is populated with flight instrumentation and pilot controls, including the overhead 
subsystem panels, to replicate the B-757 aircraft. The collimated out-the-window (OTW) scene is produced by an Evans 
and Sutherland ESIG 4530 graphics system providing approximately 200 degrees horizontal by 40 degrees vertical field- 
of-view at 26 pixels per degree. The forward windows were masked for this experiment but the side windows were 
unobstructed to allow testing of the effects of peripheral cues during approach, landing, taxi, and departure operations. 



Figure 2. Integration Flight Deck simulation facility with HUD, XVS display and head-down Research Display (RD). 

The EP occupied the left seat, as the Pilot Flying (PF). The left seat included an overhead HUD projection unit, a 20.1 
inch diagonal Liquid Crystal Display (referred to as the XVS display) and a head-down research display (RD) (Fig. 2). 
The right seat was occupied by a principal investigator (PI) who acted as First Officer during data collection. The PI 
used his/her side windows to aid the EPs during surface operations by alerting them of upcoming right turns. 

2.2.1. Head-Up Display 

The HUD was collimated and subtended approximately 26° horizontal by 21° vertical field of view (FOV). Note that to 
maintain conformality with the outside world, the FOV for the HUD imagery was fixed and could not be varied by the 
EP. The HUD presentation was written strictly in monochromatic raster format from a video source (RS-343) input. 
The input consisted of a video mix of symbology and fused synthetic/enhanced vision (S/EV) imagery. The symbology 
included “haloing” to ensure that the symbology was highlighted against the scene imagery background. Overall HUD 



brightness and contrast controls were provided to the pilot. In addition, the EP was able to independently adjust the 
flight symbology brightness relative to the raster imagery. The pilot also had a declutter control, implemented as a four- 
button castle switch on the left hand horn of the PF yoke. Four “declutter” states were available to the EP: 1) 
Symbology toggle (on/off); 2) Imagery toggle (on/off); 3) All decluttered (no symbology or imagery); and 4) All 
symbology and imagery restored. 

2.2.2. external Visibility System Display 

The XVS display subtended approximately 44° horizontal by 34° vertical FOV and was located approximately 19 inches 
from the pilot design eye point. The imagery on the XVS display was conformal with the OTW view just as the HUD 
imagery. However, this display differed from HUD as it was larger, used color, and was not collimated. The XVS 
display emulated a camera view mounted on the outside of the aircraft with flight symbology overlaid on the scene. 
Thus, any items (e.g., traffic, approach lighting system, terrain, runway markings, etc.) that would be visible to a real 
camera system would be visible in the color camera imagery. For experimental purposes, this photo-realistic camera 
imagery was unaffected by the outside weather to test for display size and peripheral cues effects. The same declutter 
control described in the HUD section above was utilized with the XVS display. 

2.2.3. Head-Down Research Display 

A head-down RD was installed over the normal instruments on the left hand side of the IFD cockpit (Fig 2). The RD 
covered the captain’s displays normal for a Boeing 757-200. The RD used an 18.1-inch diagonal case containing two 
high brightness LCDs. The two separate LCD panels, each with XGA (1024x768) resolution, rendered a Size D (6.4 
inch square viewable area) primary flight display (PDF) and navigation display (ND). 


2.3. Symbology 

The same symbology set was used for the XVS and HUD concepts (Fig. 3). The symbology included pathway guidance 
and a runway outline. The pathway symbology 7 ended at 500 ft HAT and was replaced by a runway outline and a 
glideslope reference line. A runway outline symbol (8000 ft x 200 ft) was drawn using the threshold coordinates of the 
landing runway and the aircraft navigation solution to conformally position the symbol. A glideslope reference line was 
drawn at a descent angle of 3. 1 degrees. Also, radar altitude was shown digitally underneath the flight path marker when 
below 500 ft above ground level (AGL). 


A pitch-roll guidance cue (“ball”) used modified pursuit guidance 8 along the desired path. Horizontal and vertical 
position of the ball reflects the track and flight path angles to fly to the center of the desired path. The path deviation 
indicators showed angular course deviation (i.e., glideslope and localizer-like) conditions by converting the linear path 
error data to angle errors and scaling in “dots.” Glideslope and localizer raw data indicators which included a deviation 
scale and angular deviation indication were also provided (i.e., glideslope and localizer deviation). 



Three head-up flight display concepts were evaluated by the EPs while flying approaches to Runway 16R at the Reno- 
Tahoe International Airport (airport identifier RNO). The head-down PFD and ND formats were invariant. 





Three additional head-up display conditions were tested but not reported herein. These conditions were created by 
introducing a pitch bias which simulated an increased angle-of-attack. The nominal cases reported herein used a HUD 
pitch bias of 3 degrees which was tailored to the B-757 simulator and its nominal operating approach and landing 
conditions. The additional head-up display conditions used an 1 1 degree pitch bias to approximate that of a supersonic 
transport aircraft on approach. 

2.4.1. Head-bp Flight Display Concepts 

In Figure 4 (left picture), the HUD concept is shown for the low angle of attack (AOA) condition reported herein. The 
XVS display was turned off when the EP was evaluating a HUD concept. Note that the forward windows were masked 
for the HUD concept. 

Two XVS display concepts (fig. 4, center and right pictures) were tested. The two XVS display concepts were created 
by simulating two different FOVs: 1) either the 26 x 21 degrees - identical to the HUD FOV; or, 2) the full FOV of the 
XVS display (44 x 34 degrees). The symbology used in the XVS concepts was identical to that used in the HUD 
concept. 

Simulated color camera imagery was mixed with the symbology and shown conformally on the color, uncollimated XVS 
display for both FOVs. The HUD was stowed to preclude blocking or distortion of the pilot’s forward view when using 
the XVS display. In Figure 4. both XVS display concepts are shown for the low AOA pitch attitude reported herein. 



Figure 4. Head-up display fomiats - HUD (left), XVS with HUD FOV (center), and XVS with full FOV (right). 

2.4.2. Head-Down Flight Display Concepts 

The PFD and ND closely resembled typical transport aircraft equipage for glass cockpits. However, guidance 
information was purposely removed from the PFD so that the EPs would focus on the head-up flight display concepts. 
The ND showed the RNO Runway 16R approach path, but it did not include any Enhanced Ground Proximity Warning 
System nor Traffic Alert and Collision Avoidance System information. 

2.5. Approach Lighting System Configurations 

Three different approach lighting system (ALS) configurations were used for this experiment. The first ALS 
configuration (hereinafter referred to as the VFR ALS) was representative of lights found at a VFR runway and included 
Runway Edge Identification Lights (REIL), precision approach path indicator lights (PAPI), partial threshold lights, and 
medium intensity runway lights (MIRL). The second ALS configuration (hereinafter referred to as the MALSR ALS) 
was representative of a Category I/Typc I operations runway and included MALSR (medium intensity approach lighting 
system; runway alignment indicator lights), PAPI, full threshold lights, and MIRL. The third ALS configuration 
(hereinafter referred to as the ALSF-2 ALS) was representative of a standard Category II/Type II operations runway and 
included ALSF-2 (high intensity approach lighting system with sequenced flashing lights), touchdown zone and 
centerline lighting, PAPI, full threshold lights, and high intensity runway lights (HIRL). 

2.6. Evaluation Task 

The approach task mimicked an existing visual arrival procedure reflecting an efficient and preferred routing for air 
traffic control and noise abatement. This approach requires visual meteorological conditions for the crew to see and 




avoid terrain, traffic, and obstacles while navigating with respect to ground references. The approach was a curved, 
descending path around terrain and obstacles and, thus, tests the ability of the display concepts to support this type of 
equivalent visual operation. The weather consisted of altitude-based cross winds (wind direction and intensity was 
dependent on altitude), light turbulence, and varying visibility levels (3 miles or 200 ft runway visual range, RVR). The 
EP hand-flew the base and final legs of the visual arrival to RNO Runway 16R, using the HUD or XVS display concept 
with autothrottles engaged, holding 132 knots. The aircraft was configured for landing (landing gear down and flaps 30 
degrees) prior to each run, and the aircraft was “cleared to land”. The path converged into the instrument landing system 
approach course, nominally resulting in a stabilized approach no lower than 1,000 ft HAT. EPs were instructed that the 
approach runs would end with a full stop but that a go-around should be performed if they felt the approach or landing 
was not safe. The pilot was instructed to follow a pre-briefed taxi clearance requiring the aircraft to exit the runway on a 
high-speed turnoff onto Taxiway November, turn right on Taxiway Alpha, cross over Runway 7/25, and then turn left on 
Taxiway Lima where the run ended. 

EPs also performed departures flying the RNO “Mustang 7” Departure Procedure. They maintained the runway heading 
of 168 degrees until waypoint RIJTU (about 5 nmi from the departure runway) and then turned left direct toward the 
Mustang VORTAC, where the run ended. The weather consisted of altitude-based cross winds (wind direction and 
intensity was dependent on altitude), light turbulence, and varying visibility levels (3 miles or 200 ft RVR). The EP 
hand-flew the departure with the HUD, XVS with HUD FOV or XVS with Full FOV display concept and was instructed 
to climb to 10,000 ft mean sea level (MSL) and 250 knots. 

There were up to 3 transport-sized aircraft in the runway environment, but they did not provide any conflicts for the 
ownship during approach, landing, taxi, or departure operations. There was no ATC involvement in the tasks. 

2.7. Pilot Procedures 

Since only pilot-flying evaluations were being conducted, automatic aural altitude call-outs ( e.g ., 1000, 500, 100 feet, 
etc.) were included in the simulation to “assist” in altitude awareness. Unlike current FAA regulations when landing 
with EV imagery on a HUD, for this experiment, the EP was not required to see using natural vision the required landing 
visual references (as per FAR §91.175) by 100 ft HAT. The EP was instructed to continue to landing if the required 
landing visual references were seen in the imagery on the HUD or XVS and if the EP determined that a safe landing 
could be performed. Otherwise, a missed approach should be executed. 

The EPs were instructed to fly the aircraft as if there were passengers aboard, fly the center of the approach path (within 
± Vi dot for desired performance and within ± 1 dot for adequate performance), and land as close as possible to the 
centerline and aim point (1000 feet from the threshold). After landing, they were to capture the center line and then, taxi 
at a speed with which they were comfortable using the pre-briefed taxi clearance. They were also instructed to initiate a 
go-around if the landing was not safe or if there were any safety concerns during the approach. EPs were instructed to 
stop the aircraft if they felt unsafe during surface operations. 

Prior to run commencement, the EP was briefed on the type of run to be completed, the display concept to be evaluated, 
the visibility level, and the wind magnitude and direction. 

2.8. Experiment Matrix 

Nominally, ten training runs and twenty-seven experimental runs were completed by each EP. Of the 27 experimental 
runs, 5 non-normal runs were included to investigate the pilot's awareness and reaction to unexpected events and 
conditions (e.g., failures). The non-normal data are critical determinants in the underlying safety of all-weather 
operations. Due to paper page-limit constraints, these data and also six of the 22 nominal runs ( i.e ., the so-called “high 
angle of attack” approaches) are not reported herein. 

For approach and landing runs, the experiment matrix (Table 1) consisted of a full-factorial combination of display type 
(HUD, XVS with HUD FOV, or XVS with Full FOV) and peripheral cues in side windows. The peripheral cues were 
either absent, simulating Instrument Meteorological Conditions (IMC) of 200 RVR, or present, simulating Visual 
Meteorological Conditions (VMC) of 3 miles visibility. The MALSR ALS was used for these approach and landing 
runs. 

Four additional runs were conducted to test for display type (HUD or XVS with Full FOV) and ALS (VFR, ALSF-2) 
effects. The IMC visibility level was used for these runs. These four runs are compared with 2 runs (HUD and XVS 
with Full FOV using MALSR ALS) from the Table 1 experimental matrix to test for ALS effects. 



For departure runs, the experiment matrix (Table 1) consisted of a full-factorial combination of display type and 
peripheral cues in the side windows for a total of 6 runs. 


Table 1 . Approach and departure run matrix, 
(evaluated using MALSR approach lighting system) 



HUD 

XVS with HUD FOV 

XVS with Full FOV 

IMC Approach and Landing 

V 

V 

✓ 

VMC Approach and Landing 

V 

V 

✓ 

IMC Takeoff 

V 

V 

✓ 

VMC Takeoff 

V 

V 

✓ 


Note: VMC is visual meteorological conditions and IMC is instrument meteorological conditions. 


A significant component of the test, in addition to the nominal runs, was the investigation of the ability of the EP to 
recognize and properly handle non-normal events. The number of non-normal scenarios was designed to avoid 
expectancy on the part of the flight crew. 11 Five non-normal runs were flown by each EP. The non-normals were two 
flight director failures (annunciated and unannunciated), two flare cue failures (annunciated and unannunciated), and one 
rejected takeoff. The EPs were trained to recognize the annunciated flight director failure, annunciated flare cue failure, 
and rejected takeoff. For the failures on approach, they were instructed to continue the landing if they felt it was safe to 
do so. For the rejected takeoff, they were instructed to stop the aircraft as quickly as possible on the runway. For the 
annunciated failure runs, the affected symbol (flight director or flare cue) was removed from the display and a failure 
flag was presented on the display. For the unannunciated failure runs, the affected symbol was frozen on the display and 
no failure flag was presented on the display. The rejected takeoff run was caused by a simulated right engine fire 
annunciated aurally. A between-subjects design was employed for the non-normal approach and departure runs. As 
mentioned before, the results of the non-normal runs are not reported herein but will be reported elsewhere. 

2.9. Measures 

During each approach and landing run, path error, pilot control inputs, and touchdown performance (sink rate at 
touchdown, distance fore or aft of touchdown zone, and distance left or right of centerline) were measured for analysis. 
During taxi operations, centerline tracking and taxi speed were measured. For departure runs, centerline tracking, 
heading and climb rate maintenance, and altitude capture were measured. 

After each run, pilots completed a run questionnaire consisting of the NASA Task Load Index (TLX) workload rating 9 , 
Situation Awareness Rating Technique (SART) 9 , and six Likert-type (5-point) questions specific to different constructs 
of making a stabilized and safe approach to landing, taxiing (when appropriate), or departure. 

After data collection was completed, pilots were administered two paired comparison tests: the Situation Awareness - 
Subjective Workload Dominance (SA-SWORD) 10 and Subjective Workload Dominance (SWORD) 9 techniques. The 
pilots also completed a post-test questionnaire to elicit comments on using the different display concepts, with and 
without peripheral cues, for conducting 1) approaches without a visual segment, 2) surface operations and 3) departures. 

2.10. Test Conduct 

The subjects were given a 1-hour briefing to explain the experiment purpose, F1UD and XVS concepts, pilot procedures, 
and the evaluation tasks. After the briefing, a 1-hour training session in the IFD was conducted to familiarize the 
subjects with the aircraft handling qualities, display symbologies, pilot procedures, and controls. The annunciated flight 
director and flare cue failure runs and rejected landing runs were discussed and trained. The pilot’s responsibility for 
maintaining safe operations at all times was stressed. Data collection lasted approximately 4.5 hours and was followed 
by debriefings which included the SWORD/SA-SWORD paired comparisons tests and a final questionnaire. The entire 
session including lunch and breaks lasted approximately 8 hours. 

3. RESULTS 

The objective and subjective results are presented for the low AOA display simulation conditions only. The comparison 
of high and low AOA effects will be reported elsewhere. 

For the objective performance measures and post-run questions, analysis of variance (ANOVA) analyses were conducted 
for the factors of display type (F1UD, XVS with FIUD FOV, XVS with full FOV) and peripheral cues (absent, present). 




For the post-test paired comparisons, simple ANOVAs were conducted for display type. When necessary, Student- 
Newman-Keuls (SNK) post-hoc tests with a set at 0.05 were performed. 

3.1. Objective approach standards analysis 

The Joint Aviation Authorities (JAA) Joint Aviation Requirement (JAR) All Weather Operations (AWO) performance- 
based approach standard for go-around rate (AWO-202) in approaches with decision heights below 200 ft and down to 
100 ft was applied in the objective data analysis 12 . Specifically, the standards specify that no more than 5% of the 
approaches will have localizer tracking greater than 1/3 dot or glideslope tracking greater than 1 dot between 300 ft and 
100 ft HAT for certification acceptance. These standards were not written specifically as quantitative performance 
standards for advanced vision systems (S/EV and XVS) operations, but are applied herein for comparative purposes. 

The Continuous Method 12 technique was employed to calculate the probability of success, P(a), of meeting the AWO 
exceedance criteria (1/3 dot localizer, 1 dot glideslope) with required levels of confidence with the different display 
concepts flown. The probabilities of success for meeting the AWO localizer and glideslope criteria are shown, broken 
down by display configuration and visibility condition, in Table 2. 


Table 2. Probabilities of success in meeting the AWO localizer and glideslope criteria without a visual landing segment. 


Display 

Weather 

Localizer P(a) 

Glideslope P(a) 

Number of Runs 

Number of Go-Arounds 

HUD 

VMC 

100.0 

78.2 

24 

0 

HUD 

IMC 

100.0 

67.4 

24 

0 

XVS HUD FOV 

VMC 

100.0 

94.3 

24 

0 

XVS HUD FOV 

IMC 

100.0 

80.6 

24 

1 

XVS Full FOV 

VMC 

100.0 

96.0 

24 

0 

XVS Full FOV 

IMC 

100.0 

92.6 

24 

0 


Note: Approaches flown using low angle of attack pitch attitude and MALSR approach lighting system. 


The data in Table 2 shows that localizer tracking was maintained, irrespective of the display being flown or the absence 
or presence of peripheral cues in the side windows. However, glideslope tracking was better under the following 
conditions: 1) when flying with the larger FOV display concepts compared to the smaller (HUD) FOV display concepts, 
2) with the color XVS HUD FOV display concepts compared to the monochrome HUD display concepts, and 3) when 
peripheral cues were present for the display concept. The only display concept successfully meeting the JAR AWO-202 
localizer and glideslope criterion (greater than 95% of the time) was the XVS Full FOV display concept flown with 
peripheral cues present in the side windows. This color, uncollimated, display concept showed conformal imagery over 
a 34° vertical FOV. 

The Continuous Method was applied to data from a prior experiment 12 for comparison. In this experiment, pilots 
evaluated the use of SV on a HUD during an instrument approach but they were required to transition to a natural vision 
segment, just as they would have to do today under existing operating regulations. (Note that the symbology used during 
the SV-HUD experiment was identical to that used in the current experiment, but the approach was flown at 138 knots 
instead of 132 knots.) The analysis revealed that 100% of the runs met the localizer criteria but only 60% of the runs 
met the glideslope criteria. The comparison of the two experiments shows that slight improvements in glideslope 
tracking may be realized with elimination of the visual segment (60% vs. 67%). The differences are not large, but the 
trends make intuitive sense. By allowing the pilot to concentrate on the task and display information, improved flight 
tracking performance results. The visual segment transition forces the pilots to simultaneously perform glideslope 
corrections and acquire the required landing visual references through normal vision. 

3.2. Objective landing standards analysis 

Existing JAR AWO 12 performance -based landing standards (AWO 131) for longitudinal position and lateral position 
from centerline were applied in the objective landing data analysis. Specifically, the standards state that no longitudinal 
touchdown earlier than a point on the runway 200 ft from the threshold or beyond 2700 ft from the threshold and no 
lateral touchdown with the outboard landing gear more than 70 ft from the runway centerline to a probability of 1 x 10" 6 . 
These standards pertain to the general concept of low-visibility approach and landings, but were not written specifically 
for operations with advanced vision systems such as S/EV or XVS. 




This experiment used an aim point located 1000 ft from runway threshold. For the simulated 757 aircraft, the outboard 
landing gear would be 70 ft from the centerline when the fuselage (the recorded lateral landing position reported herein) 
is at 58 ft lateral deviation from centerline, assuming no crab angle at touchdown. 

In Figure 5, the touchdown data are shown, broken out by display concept but collapsed across peripheral cue influence. 
Included on this plot (in red, dashed rectangle) is the +/-58 ft lateral and 200 ft to 2700 ft longitudinal touchdown zone 
definition used in the JAR AWO landing criteria. 

Statistical analysis against the AWO landing criteria meeting the lxlO" 6 probability requirements has not been 
completed, but visual inspection of the data shows that both XVS display concepts regardless of their FOV were within 
the JAR AWO touchdown zone definition. 
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Figure. 5. Landing performance data for HUD (top), XVS with HUD FOV (middle), and XVS with large FOV (bottom). 

3.3. Mental Workload 

Mental workload was assessed after each experimental run, using the NASA TLX, and post-test, using the SWORD 
technique. 

NASA TLX is a multi-dimensional rating procedure that derives an overall workload score based on a weighted average 
of ratings on six subscales (mental demand, physical demand, temporal demand, performance, effort, and frustration 
level). The NASA TLX subscales went from 0 (Low) to 100 (High) for the workload ratings. TLX ratings were 
provided for every data run and configuration. 

Pilots were administered the paired-comparison SWORD scale that enabled comparative ratings of mental workload. 
Mental workload was defined for the pilots as “the amount of cognitive resources available to perform a task and the 
difficulty of that task.” The pair-comparison test was structured to compare the effects of peripheral cue (i.e., 
“visibility”) and display type (i.e., VMC/HUD, IMC/HUD, VMC/XVS HUD FOV, IMC/XVS HUD FOV) flown by the 
EP. Note that this comparison does not include the effects of FOV. Only the HUD FOV configurations were 
considered. 


3.3.1. NASA TLX 

Approach and Landing Operations : An ANOVA revealed that display (F(2,142)=9.64, p<0.001) and peripheral cues 
(F(l,142)=9.02, p=0.003) were highly significant for workload ratings. Post-hoc tests showed two unique subsets for 
workload ratings with the display concepts: 1) XVS with Full FOV (mean=30) having lower workload and 2) XVS with 
F1UD FOV (mean=34) and FIUD (mean=34) having higher workload. Pilots rated their workload significantly lower 
when they had peripheral cues (mean=30) on the approach compared to when they didn’t have them (mean=33). 

Surface Operations : An ANOVA revealed that display (F(2,139)=28.75, p<0.001) and peripheral cues (F(l,139)=4.86, 
p — 0.029) were significant for workload ratings. Post-hoc tests showed three unique subsets for workload ratings: 1) the 
XVS with Full FOV concept (mean=24) having the lowest workload, 2) the XVS with FIUD FOV concept (mean=28), 
and 3) the FIUD concept (mean=33) with the highest workload during surface operations. Pilots rated their workload 
significantly lower when they had peripheral cues (mean=27) while taxiing compared to when they did not have them 
(mean=29). 

Departure : An ANOVA revealed that the peripheral cues factor (F(l,46)=6.01, p=0.018) was significant for pilot 
departure workload ratings, but display (mean=29) was not significant (p>0.05) for this measure. Pilots rated their 
workload lower when they had peripheral cues (mean=28) in the side windows during a departure compared to when 
they didn’t have them (mean=30). 

3.3.2. SWORD 

The post-test paired-comparison SWORD data indicate that visibility/display type (F(3, 92)=49.05, p<0.001) was highly 
significant for the pilot ratings of mental workload. Post-hoc tests (SNK using a=0.05) showed three unique subsets for 
the mental workload ratings with the 4 visibility/display combinations: 1) VMC/XVS FIUD FOV (lowest workload); 2) 
IMC/XVS HUD FOV and VMC/HUD; and 3) IMC HUD (highest workload). 

3.3.3. Workload Discussion 

The subjective post-run ratings indicated the workload was significantly reduced when using the larger displays ( i.e ., 
greater FOV) and when peripheral cues were available OTW for approach and surface operations. In consideration of 
the mean rating differences however, the workload changes created by these effects are not operationally significant. In 
all cases, the pilot workload was very manageable when performing approach and landing using any of the evaluated 
concepts. This result is somewhat surprising for surface operations, since, in IMC conditions (200 RVR) and with the 
smaller FOV displays in particular, usable visual information is provided to the pilot only in very limited areas (the HUD 
and side windows). These surface operations results were likely influenced by the PI aiding the EP during right taxi 
turns. 

During departures, no workload differences were found for display size {i.e., FOV) but there were workload 
improvements when peripheral cues were added. Even though statistically significant workload differences were found, 
operationally these differences were not substantial - all were ranked at a moderately low workload level on the TLX 
scale. 

The subjective post-test ratings where display size was held fixed revealed that having color in the display was a larger 
influence in workload ratings than a collimated display. These results were supported by subjective commentary made 
during the post-test interviews. 

3.4. Situation Awareness 

Situation awareness was assessed after each experimental run, using the post-run SART, and post-test, using the SA- 
SWORD measures. 

SART is a multi-dimensional rating technique using the constructs of: 1) demand on attentional resources; 2) supply of 
attentional resources; and, 3) understanding. From these components, the SART rating is “understanding” reduced by 
the difference of “demand” minus “supply” {i.e., SART = {(understanding) - (demand - supply)}). SART ratings were 
provided for every data run and configuration. 

Similar to the SWORD described above, the SA-SWORD is a paired-comparison technique that provides relative 
situation awareness ratings. For these comparisons, SA was defined as “the pilot’s awareness and understanding of all 
factors that will contribute to the safe flying of their aircraft under normal and non-normal conditions.” The pair- 
comparison test was structured to compare the effects of peripheral cue {i.e., “visibility”) and display type {i.e.. 



VMC/HUD, IMC/HUD, VMC/XVS HUD FOV, IMC/XVS HUD FOV) flown by the EP. Note that this comparison 
does not include the effects of FOV. Only the HUD FOV configurations were considered. The SA-SWORD measure 
differs from the post-run SART measure construct as it is a pair-comparison test, it was administered post-test for all 
configurations, and the underlying definition and construct for the SA ratings are different. 

3.4.1. SART 

Approach and Landing Operations : An ANOVA revealed that display (F(2,142)=10.94, p<0.001) and peripheral cues 
(F(l,142)=15.66, p<0.001) were highly significant for pilot approach SART ratings. Post-hoc tests showed two unique 
subsets for SA ratings with the display concepts: 1) XVS with Full FOV (higher SA) and 2) XVS with HUD FOV and 
HUD (lowest SA). Pilots rated their SA significantly higher when they had peripheral cues on the approach compared to 
when they didn’t have them. 

Surface Operations : An ANOVA revealed that display (F(2,139)=47.46, p<0.001) was highly significant for pilot taxi 
SART ratings, but peripheral cues was not significant (p>0.05) for this measure. Post-hoc tests showed three unique 
subsets for the SA ratings with the XVS with Full FOV concept providing the highest SA, followed by the XVS with 
HUD FOV concept, and then finally the HUD concept providing the least SA during surface operations. 

Departure : An ANOVA revealed that display (F(2,46)=4.07, p=0.024) was significant for pilot departure SART ratings, 
but peripheral cues was not significant (p>0.05) for this measure. Post hoc tests showed that the XVS with Full FOV 
concept provided higher SA than either the XVS with HUD FOV concept or HUD concept during a departure. There 
were no appreciable SA differences between these two smaller size display concepts. 

3.4.2. SA-SWORD 

The post-test SA-SWORD data indicate that visibility/display type (F(3, 92)=96.79, p<0.001) was highly significant for 
the pilot ratings of SA. Post-hoc tests (SNK using a=0.05) showed four unique subsets for the SA ratings with the 4 
visibility/display types: 1) VMC/XVS HUD FOV (highest SA); 2) IMC/XVS HUD FOV; 3) VMC/HUD; and 4) IMC 
HUD (lowest SA). 

3.4.3. Situation Awareness Discussion 

The subjective post-run ratings indicated that SA was significantly enhanced when using the larger displays compared to 
the smaller ones for approach and landing, surface, and departure operations. SA improvements due to having 
peripheral cues in the side windows was only realized during approach and landing operations. This result for surface 
operations is somewhat counter-intuitive given the importance of full-field of regard information for taxi path and traffic 
awareness on the airport but they, too, were probably influenced by the PI aiding the EP during surface operations. 

Similar to the workload results, the subjective post-test ratings where display size was held fixed revealed that having 
color in the display was a larger influence on SA than a collimated display. These results were also supported by pilot 
comments during the post-test interviews. 


4. CONCLUSIONS 

An experiment was conducted to investigate the use of S/EV and XVS technologies as enabling technologies for future 
all-weather operations. The experimental objectives were to determine the feasibility of these advanced vision 
technologies to provide for all weather (visibility) landing capability without the need (or ability) for a visual approach 
segment and to determine the interaction of S/EV and XVS and peripheral vision cues for terminal area and surface 
operations. 

Objective results indicate that elimination of the visual segment of the approach had no adverse affects on localizer 
tracking as it was excellent regardless of display size (large FOV or HUD FOV) or whether the pilot had peripheral cues 
or not. Elimination of the visual segment appears to have improved a pilot’s glideslope tracking compared to previous 
experiments requiring a visual segment. This improvement can most likely be attributed to the fact that the pilots didn’t 
have to simultaneously perform glideslope corrections and find required visual landing references in order to continue a 
landing. In addition, a larger display size appears to offer slight improvements in glideslope tracking over smaller size 
displays. Landings were completed safely regardless of display type. In fact, landings made with either color XVS 
display concept were within JAR AWO touchdown point criteria. 



Not surprising, subjective results indicate that a larger display size provides significantly more situation awareness 
during approach, surface, and departure operations than smaller size displays. Workload reductions during approach and 
landing operations were also found with the larger size displays compared to the smaller ones, but not during departure 
operations. Having peripheral cues in the side windows significantly reduced pilot workload during all operational 
phases tested, but these workload differences appear operationally inconsequential. Similar results for peripheral cues 
were found for SA. These results are somewhat surprising given the importance of full field-of-regard information for 
the surface operations task (e.g., awareness and maintenance of the taxi path; separation from other traffic; and ground 
hazards). Further investigation of these results is necessary. 

When display size was held fixed, SA and workload improvements were seen with the color, uncollimated XVS display 
compared to the monochrome, collimated HUD display, according to paired-comparison tests. Subjective comments 
indicate that color was the biggest discriminator, not collimation, in these pilot ratings. This result is not surprising 
considering that a HUD is designed to operate in concert with the out the window view. Since the forward vision was 
obscured, the XVS provided this missing information. 

The data showed little evidence of a display collimation effect ( i.e ., when comparing XVS-HUD vs. HUD). In this 
experiment, this effect was confounded by the display color differences, but there was little subjective commentary 
noting an influence of display collimation. Use of a motion-based simulator while conducting approach, landing and 
surface operations with these conformal head-up display concept using advanced vision technologies should be explored 
to evaluate the effects of motion on collimation, groundspeed, and ground rush cues. 
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